Feature extraction through local learning

RELIEF is considered one of the most successful algorithms for assessing the quality of features. It has been recently proven that RELIEF is an online learning algorithm that solves a convex optimization problem with a margin-based objective function. Starting from this mathematical interpretation, we propose a novel feature extraction algorithm, referred to as LFE, as a natural generalization of RELIEF. LFE collects discriminant information through local learning and can be solved as an eigenvalue decomposition problem with a closed-form solution. A fast implementation of LFE is derived. Compared to PCA, LFE also has a clear physical meaning and can be implemented easily with a comparable computational cost. Compared to other feature extraction algorithms, LFE has an explicit mechanism to remove irrelevant features. Experiments on synthetic and real-world data are presented. The results demonstrate the effectiveness of the proposed algorithm. Statistical Analysis and Data Mining Submitted in May 2008, revised in September 2008, accepted in January 2009 ∗This work was supported in part by the Komen Breast Cancer Foundation under grant No. BCTR0707587. Please address all correspondence to: Dr. Yijun Sun, Interdisciplinary Center for Biotechnology Research, University of Florida, P.O. Box 103622, Gainesville, FL 32610-3622, USA. E-mail: [email protected].